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Introduction 


The  second  term  of  the  grant  has  continued  to  focus  on  identifying  key  differentiating 
performance  factors  in  the  pre-mastery  and  mastery  phases.  Identifying  these  factors  was 
primarily  done  by  evaluating  survey  results  from  the  first  round  of  data  collection,  and  analyzing 
performance  on  two  stations.  The  lab  was  able  to  compare  performance  for  current  study 
participants  at  one  station  to  those  whose  data  was  analyzed  from  previously  recorded  simulation 
based  LVH  repairs.  Station  design  has  been  optimized  to  allow  for  the  best  possible  data 
collection. 


Summary  for  Statement  of  Work  Progress 

The  following  section  details  each  element  of  the  SoW  as  it  has  been  addressed  through 
our  work  thus  far.  For  review,  the  following  four  objectives  guided  this  work: 

Objective  One:  To  evaluate  mental  rehearsal  as  an  intervention  for  skill  decay  in  the  pre¬ 
mastery  phase. 

Objective  Two:  To  identify  key  differentiating  performance  factors  for  the  pre-mastery  and 
mastery  phases. 

Objective  Three:  To  develop  a  generalizable,  multi-variable,  predictive  model  of  skills  decay. 
Objective  Four:  To  develop  an  efficient  and  effective  set  of  assessment  tools  and  individualized 
training  recommendations  to  counteract  skills  decay. 

In  its  second  year,  our  work  has  continued  to  focus  on  Objectives  Two  and  Four  because 
significant  progress  on  these  two  objectives  is  required  in  order  to  address  Objectives  One  and 
Three.  We  intend  to  continue  refining  the  elements  of  Objectives  Two  and  Four  over  the  next 
year  while  our  primary  focus  will  shift  towards  addressing  Objectives  One  and  Three  as  data 
analysis  continues. 

Objective  One 

No  progress  at  the  time  of  this  report.  We  plan  to  address  this  objective  as  data  collection 
continues. 

Objective  Two 

The  greatest  area  of  progress  for  Objective  Two  has  been  compiling  the  data  collected  in 
2014  and  conducting  analyses.  During  data  collection  common  error  checklists  were  completed, 
and  the  checklists  were  later  used  to  code  and  timestamp  key  events  or  errors  using  video 
collected  from  each  clinical  scenario  station.  Videos  from  each  station  were  also  coded  for  the 
major  steps  that  occurred  in  each  participant’s  procedure.  Additionally,  standardized  final 
product  images  of  the  artificial  skin  from  the  laparoscopic  ventral  hernia  (LVH)  station  were 
obtained  (Figures  1-2).  The  images  were  taken  from  a  standard  height  and  position  so  that  each 
image  could  be  analyzed  for  the  two-dimensional  location  of  stitches  or  ports  placed  by  the 
participant. 
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Quantifying  laparoscopic  port  placement  using  position  clustering 

Participants  indicated  laparoscopic  port  placements  on  surveys  administered  at  data  collection 
sessions.  The  team  later  analyzed  the  data  to  calculate  the  distances  from  the  selected  placements 
of  ports  to  the  edges  of  the  hernia  defects  (Figure  3).  An  abstract  was  submitted  to  the  Academic 
Surgical  Conference  (ASC)  (Rutherford,  D.N.,  D’Angelo,  A.D.,  Kwan,  C.,  Barlow,  P.B.,  & 

Pugh,  C.M.,  2014)  and  was  accepted  for  a  podium  presentation.  Our  data  indicated  a  significant 
difference  in  the  variances  of  placements  for  four  different  hernia  types.  Residents  had  the  least 
amount  of  variance  for  an  epigastric  hernia  and  the  most  variance  within  an  inguinal  hernia. 


Port  Distance  to  Hernia 

25 

■  5  mm  Ports 


Epigastric  RLQ  Incisional  Right  Inguinal  Right  Inguinal  Right  Inguinal  Right  Inguinal  Umbilical 
(ALL)  (OTHER)  (TAPP)  (TEP) 

(N  =  6)  (N  -20)  (N  - 20)  (N  ^  12)  (N=7)  (N  =  l)  (N  =  14) 

•  WQ  •  Higl't  lower  Quadrant 


Figure  3:  Results  from  hernia  port  placement  surveys.  Distances  of ports  from  the 
edge  of  the  hernia  defect  are  listed  by  hernia  and  procedure  option. 


Perception  of  Skill  Reduction  during  Dedicated  Research  Time 

At  data  collection,  participants  rated  their  perceived  levels  of  difficulty  with  and  confidence  in 
performing  the  simulated  clinical  scenarios,  as  well  as  the  perceived  reduction  in  their  skills  with 
time  spent  away  from  clinical  practice  in  a  lab  setting.  The  team  later  analyzed  the  survey  results 
and  submitted  an  abstract  to  ASC  (Ray,  R.D.,  Barlow,  P.B.,  D’Angelo,  A.D.,  Pugh,  C.M.,  2014). 
The  abstract  was  accepted  for  a  podium  presentation  at  the  meeting  and  a  corresponding 
manuscript  is  under  review  the  Journal  of  Surgical  Research  (D’Angelo,  A.D.,  Ray,  R.D., 
Jenewein,  C.G.,  Jones,  G.F.,  Pugh,  C.M.,  Under  review).  Results  showed  that  residents  who  were 
less  confident  in  performing  the  presented  procedures  also  perceived  their  skills  would  be  more 
greatly  reduced  by  time  spent  in  the  lab  (Figure  4).  However,  this  was  not  true  for  the 
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laparoscopic  ventral  hernia  (LVH)  station;  residents  may  perceive  their  initial  skills  to  be  so  low 
that  time  in  the  lab  would  not  reduce  skills  further. 


Table  X.  Pairwrse  Companions  of  Participants'  Perceived  Difficulty,  Confidence,  and  Skills  Reduction 
Rat  ngs 


Mean  (SD) 

LVH 

Bowel 

Anastomose 

Subclavian  Central 

Line  Insertion 

Unnary  Catheter 
Placement 

Perceived 

Difficulty* 

3.56  (1.09)*** 

3.04  (0.94)* ** 

1.96  (0.79)*  ** 

1.16  (038)*  ** 

Perceived 

Confidence6 

2.00(1.12)* 

2.32  (1.22)* 

3.56  (0.92)* 

4.24  (0.72)* 

Perceived 

Reduction 

2.75  (1.14) 

2.96  (0.98) 

2.04  (0.77)** 

1.50  (0.52)*** 

*p<-05;  "p<.01;  •**p<.001 

*5-point  tikert  scale:  l=Not  difficult,  5=  Extremely  difficult 
^•point  Likert  scale:  l^Not  confident  SsExtremely  confident 
'S-pomt  Ukert  scale:  l=No  reduction,  5=Very  large  reduction 


Figure  4:  Table  of  comparisons  of participants’  perceived  difficulty,  confidence,  and 
reduction  in  skills  for  each  of  the  simulated  clinical  procedures  performed 


Error  Tolerance  during  the  laparoscopic  ventral  hernia  repair 

In  the  first  year,  the  team  developed  and  refined  an  error  framework  for  laparoscopic  hernia 
repair  using  data  from  previously  recorded  resident  performance  on  simulation-based  LVH 
repairs  (Figure  2).  The  team  conducted  further  analyses  on  this  data  in  this  second  year. 
Coordination  errors  were  used  to  assess  the  relationship  to  total  errors  and  error  tolerance.  An 
abstract  was  submitted  and  accepted  as  a  podium  presentation  to  the  American  College  of 
Surgeons  Accredited  Education  Institutes  (ACS-AEI)  consortium  meeting  (Gwillim,  E.C., 
D’Angelo,  A.D.,  Law,  K.E.,  Cohen,  E.R.,  Rutherford,  D.N.,  Pugh,  C.M.,  2015)  with  a 
corresponding  paper  under  review  to  the  Journal  of  Surgery  (Gwillim,  E.C.,  Law,  K.E., 
Rutherford,  D.N.,  D’Angelo,  A.D.,  Minkoff,  E.C.,  Pugh,  C.M.,  Under  review).  Results  showed 
residents  committed  an  average  of  15.45  (SD=4.61)  errors  overall  and  1.70  (SD=2.25) 
coordination  errors.  There  was  a  significant  correlation  between  total  errors  and  coordination 
errors  (r(18)=0.572,  p=0.008). 


Procedure  Step 


Figure  5:  The  number  of  coordination  errors  per  procedure  step 
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Rule-based  errors  during  the  laparoscopic  ventral  hernia  repair 

The  team  compared  suture  and  fixation  errors  committed  during  the  LVH  repair.  Resident 
performance  on  the  LVH  station  was  compared  to  the  error  analyses  conducted  in  the  first  year. 
An  abstract  was  submitted  and  accepted  for  a  poster  presentation  to  the  Association  for  Surgical 
Education  (ASE)  meeting  (Law,  K.E.,  D’Angelo,  A.D.,  Ray,  R.D.,  Pugh,  C.M.,  2015).  Results 
showed  the  majority  of  junior  residents  (87.5%)  made  at  least  one  error  relating  to  transfascial 
suture  or  tacker  fixation.  Of  the  seven  errors  committed  by  the  junior  residents,  three  errors  were 
most  prevalent:  (1)  failure  to  cut  skin  prior  to  inserting  suture  passer;  (2)  use  of  same  hole  in 
peritoneum  to  pull  up  2nd  suture;  and  (3)  failure  to  tie  or  secure  sutures  prior  to  tacking 
(F(l,30)=32.69,  p<.001)  (Figure  6).  Analysis  of  the  senior  resident  data  revealed  similar  error 
types  and  error  occurrence  of  two  of  the  three  the  most  prevalent  errors. 


I  H 


Fails  to  cut  Pulls  up  two  Uses  same  hole  Fails  to  tie  Mesh  not  flat  No  counter  Tacker  slips  on 
skin  prior  to  sutures  with  in  peritoneum  down/secure  allprior  to  tacking  pressure  with  mesh 
inserting  the  suture  passer  at  to  pull  up  2nd  sutures  before  tacking 

suture  passer  the  same  time  suture  tacking 

device 

Error  Types 


■  Junior  Residents 

■  Senior  Residents 


Figure  6:  Proportion  of participants  (%)  committing  errors  during  LVH  repair 


Motor  control  in  a  simulated  task 

At  one  station,  participants  performed  a  simulated  surgery  task  by  controlling  the  movement  of  a 
handle  of  a  force-feedback  device  (Figure  7).  The  distance  of  movement  overshoot  following  the 
puncture  event  was  analyzed  based  on  task  conditions  and  with  respect  to  learning  over  the 
course  of  trials.  The  abstract  was  submitted  and  accepted  as  a  poster  presentation  to  International 
Conference  on  Intelligent  Robots  and  Systems  (IROS)  (Huang,  FC.,  2014).  Results  indicated 
participants  were  able  to  recover  performance  within  each  trial  block  despite  changes  to  tissue 
conditions.  Regression  analysis  showed  that  overshoot  distance  changed  with  respect  to  trials 
significantly  in  each  block  with  greatest  change  in  the  initial  block  (m=-1.8  mm/trials,  CI:-2.3,- 
1.2)  (Figure  8).  Secondly,  we  also  found  that  performance  was  significantly  disrupted  at 
transitions  between  blocks  only  when  the  stiffness  was  increased.  Overshoot  increased  at 
transitions  between  blocks  2-3  (mean:  8.6mm  Cl:  6.4,  10.7),  and  3-4  (change:  6.3mm  Cl:  3.0, 
9.6),  but  did  not  achieve  significance  for  blocks  1-2  (mean:  1.8mm  Cl:  -0.8,  4.4). 
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Figure  7:  Controlling  a  haptic  device,  participants  stretched  a  virtual  tissue  along  a  target  line.  The  goal  was  to 
puncture  the  tissue  while  minimizing  movement  overshoot  following  the  puncture  event. 


(A) 


(B) 


0  - I - , - , - ■ - , - , - , - ^ 

0  1  2  3  4  5 

Time  (sec) 


Figure  8:  (A)  Force  and  deflection  plots  for  one  trial  of  a  simulated  tissue  puncture  task.  The  tissues  stiffness  is 
constant  until  a  threshold  force  is  reached  at  which  point  the  force  is  immediately  zero.  The  trial  performance  was 
computed  as  the  distance  at  maximum  overshoot  with  respect  to  the  location  of  the  puncture  event.  (B)  Participants 
were  presented  with  32  trials  of  a  simulated  puncture  task,  in  8  blocks  with  varying  force  thresholds.  Overshoot 
distance  (OD)  decreased  significantly  in  each  block  (left),  as  in  indicated  by  regression  slope  values  (center).  The 
change  between  blocks  (right)  indicated  a  significant  disruption  in  performance  only  between  2-3  and  3-4.  Error  bars 
indicated  95%  CL 


Objective  three 

No  progress  at  the  time  of  this  report.  We  plan  to  address  this  objective  as  data  collection 
and  analysis  continues. 

Objective  Four 

Data  Collection  Efforts 

The  primary  focus  of  the  study  team’s  efforts  surrounded  acting  on  lessons  learned  from 
previous  pilot  studies,  training  lab  personnel,  and  streamlining  current  study  protocols  all  in 
an  effort  to  maximize  the  quality  of  the  data  collected  from  our  participants.  Most  notably, 
this  process  included  three  distinct  features: 

1)  Redesigning  the  clinical  procedure  stations; 

2)  Improving  existing  study  protocols/instructions;  and 

3)  Providing  standardized  training  and  assessment  for  all  lab  personnel  who  will  be 
involved  in  data  collection. 

Redesigning  the  Clinical  Procedure  Stations 

Based  on  feedback  from  previous  pilots,  expert  reviews  of  each  station,  and  internal 
evaluation,  each  of  the  clinical  procedure  stations  underwent  a  significant  improvement  over 
the  course  of  quarter  1 . 
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First,  noted  “experts”  on  each  of  the  four  clinical  procedure  stations  (Central  Line,  Bowel 
Anastomosis,  Urinary  Catheterization,  and  Laparoscopic  Ventral  Hernia)  were  asked  to 
provide  their  feedback  and  suggestions  for  how  each  station  may  be  improved.  For  each 
review,  the  expert  was  asked  a  series  of  semi-formal  interview  questions  by  project 
management  both  before  and  after  performing  the  procedure  (Figures  below  for  examples). 
The  audio  and  video  from  these  expert  reviews  were  transcribed  and  coded  to  generate  a  list 
of  possible  improvements  or  changes  to  the  four  stations. 

The  second  major  improvement  to  the  clinical  procedure  stations  was  consolidating  and 
simplifying  the  setup  of  each  station.  The  following  examples  show  some  of  the 
improvements  made  to  each  station. 
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Station  A:  Subclavian  Central  Line.  The  station  originally  used  a  blue  foam  wedge  that  propped  the  simulator  in  a  Trendelenberg 
position  with  the  instruments  to  the  participant’s  left.  The  expert  mentioned  it  would  be  very  unlikely  that  someone  would  organize 
their  procedure  space  in  such  a  way,  and  that  the  instruments  would  usually  be  either  on  the  patient’s  chest  or  on  a  mayo  stand  to  the 
physician’s  right.  A  wooden  fixture  was  developed  to  address  this  challenge,  so  that  the  instruments  could  be  placed  on  a  tray  in  a 
location  that  simulated  either  the  chest  or  a  Mayo  stand 


Figure  9- 10: Station  A:  Subclavian  Central  Line  Station 


Station  B:  Bowel  Anastomosis.  This  station  also  received  heavy  alterations  based  on  the  expert  feedback.  Figure  1 1  and  12  shows  the 
before  and  after  images  of  the  station  where  the  small  blue  basin  has  been  replaced  with  a  raised  (and  height  adjustable)  flat  tray.  The 
tray  also  includes  two  “clips”  that  hold  the  bowel  segment  against  the  tray  as  the  bowel  would  realistically  be  adhered  to  the 
abdominal  wall. 


Figures  11-12:  Station  B:  Bowel  Anastomosis 
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Station  C:  Urinary  Catheter.  Participants  at  this  station  require  a  variety  of  catheters  spanning  different  sizes  and  configurations.  There  are 
also  a  number  of  consumable  materials  used  during  the  procedure  such  as  lubricant  and  sterile  saline.  A  major  improvement  to  the  participant’s 
ease  of  working  in  the  space  was  the  addition  of  a  stand  fixture  to  hold  and  display  each  of  the  catheter  options.  This  provides  the  user  with  a 
means  to  see  each  option  quickly  prior  to  selecting  one  for  use  in  the  simulation.  Standardized  locations  for  the  consumables  containers  were  also 
selected  to  be  within  the  volume  captured  by  motion  sensors  and  adjacent  to  the  catheter  stand. 


Figures  13-14:  Station  C:  Urinary  Catheter 
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Station  D:  LVH.  Early  setup  of  this  laparoscopic  simulation  did  not  meet  participants’  needs  for  availability  of  all  necessary  monitor  views, 
visualization  and  identification  of  the  workspace,  and  layout  of  instruments  needed.  A  large  monitor  was  added  in  a  comfortable  viewing 
position  for  the  participant  and  researcher  assistant.  Various  placards  were  strategically  placed  in  the  simulator  environment  to  identify  key 
landmarks  and  elements  to  the  participants  for  their  orientation  to  the  operative  environment.  Standardized  sets  and  layouts  of  tools  were  selected 
and  provided  via  a  Mayo  stand  that  was  placed  in  a  location  with  comfortable  access  by  the  participant.  The  new  locations  of  simulator  elements 
and  tools  allowed  the  participant  to  make  greater  modifications  to  their  procedure,  such  as  preparing  and  placing  additional  laparoscopic  ports 
into  the  model  to  facilitate  their  performance. 


Figures  15-16:  Station  D:  Laparoscopic  Ventral  Hernia 
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Sensor-containing  Lab  Jackets.  In  pilot  testing,  it  was  identified  that  the  attachment  of  motion  capture  sensors  to  participants  presented  a  great 
challenge  for  sensor  placement.  The  attachment  of  the  sensor  wires  posed  an  ergonomic  issue  for  the  participant’s  performance.  To  minimize 
ergonomic  impacts  and  potential  catching/tripping  hazards,  lab  jackets  were  provided  to  participants  for  the  duration  of  their  simulations.  These 
jackets  had  Velcro  straps  affixed  to  the  shoulders  and  sleeves  to  facilitate  rapid  and  repeatable  attachment  of  sensors  to  individuals.  The  jackets 
also  provided  a  means  to  contain  and  transport  additional  recording  equipment  such  as  the  portable  audio  recorders  used  to  record  participants’ 
verbal  responses  at  each  procedural  station.  Participants  indicated  that  the  jackets  did  not  restrict  any  of  their  movements  and  provided  a  more 
realistic  testing  scenario. 


Figures  17-18:  Motion  Sensor  Containment 
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These  changes  eliminated  many  of  the  problems  noted  by  previous  pilot  participants  as  well  as  made  it 
so  the  study  could  feasibly  be  conducted  in  a  somewhat  smaller  space  than  was  originally  required. 

The  majority  of  these  changes  were  implemented  and  tested  at  the  third  and  final  pilot  data  collection  on 
June  6th  2014. 

Improving  Existing  Study  Protocols/Instructions 

The  second  area  where  the  research  team  focused  their  efforts  was  streamlining  and  improving  the 
protocols  and  instructions  for  how  data  were  to  be  collected  at  each  station.  These  improvements  included 
creating  dedicated  Station  Manager  Binders,  establishing  dedicated  researcher  responsibilities  for  each 
station,  and  finalizing  the  Participant  Survey  Workbook. 

The  first  two  items  in  this  list  are  improvements  to  the  protocol  on  the  researcher  side  of  the  study, 
which  were  implemented  to  improve  the  efficiency,  reliability,  and  validity  of  the  team’s  data  collection. 

For  example,  the  Station  Manager  Binders  provide  a  standardized  set  of  instructions  for  how  each  station  is 
to  be  organized,  maintained,  and  managed,  so  that  the  station  would  always  be  run  consistently,  regardless 
of  the  researcher  at  that  station.  Also,  these  binders  provide  a  centralized  location  (titled  “Station  Manager 
Notes”)  where  specific  incidents  or  phenomena  that  occur  during  a  collection  can  be  recorded  for  later 
review.  The  team  further  solidified  this  process  by  establishing  precise  researcher  responsibility  cards  for 
each  station  that  display  the  exact  order  of  tasks  each  researcher  is  responsible  for  during  a  collection. 

Similarly,  the  Participant  Survey  Workbook  received  substantial  formatting  changes  to  help  improve  the 
quality  of  data  collection  and  reduce  the  overall  demand  on  each  participant.  The  spiral-bound  workbook 
includes  clarified  instructions  and  an  easy-to-navigate  tabbing  system  for  each  of  the  post-procedure 
surveys.  Several  example  images  of  this  survey  are  included  in  the  figure  below,  and  a  PDF  copy  of  the 
workbook  was  previously  submitted  for  your  review. 
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Figure  19-20:  Final  Participant  Workbook 
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Finally,  changes  were  made  to  the  way  residents  are  tracked  over  the  course  of  their  participation  to 
further  improve  the  consistency  and  quality  of  data  while  reducing  the  demand  on  each  participant.  A 
custom  database  was  created  which  is  used  to  produce  a  series  of  individualized  participant  “Labels”  used  at 
each  collection.  As  opposed  to  hand-writing  or  using  a  notecard  to  designate  participant  ID  and  finished 
products,  the  participant  simply  hands  the  researcher  at  Station  B:  Bowel  Anastomosis,  and  Station  D:  LVH 
their  labels  which  are  then  stuck  directly  to  their  bowel  segment  container  and  simulator  skin,  respectively. 
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Providing  Training  and  Assessment  for  All  Lab  Personnel  Who  will  be  Involved  in  Data  Collection 

The  third  area  of  work  on  which  the  team  focused  was  in  providing  organized  training  for  any  of  the  lab 
personnel  who  will  be  involved  with  collecting  data.  The  individuals  who  had  been  responsible  for 
managing  each  of  the  four  clinical  procedure  stations  at  the  previous  two  pilots  were  selected  as  “Station 
Team  Leaders.”  Together  with  the  project  and  lab  managers,  these  team  leaders  developed  a  set  of 
structured  learning  objectives  and  training  activities  to  prepare  other  lab  personnel  (i.e.  “Station  Team 
Members”)  to  fill  the  role  as  primary  or  secondary  researcher  for  the  stations. 

The  station  teams  engaged  in  three  structured  training  periods:  1)  initial  training,  2)  follow-up  review 
and  discussion,  and  3)  final  assessment  performance.  The  final  assessment  for  each  of  the  stations  included 
a  standard  rubric  scoring  system  that  was  used  to  rate  each  of  the  station  team  members  in  their  ability  to 
perform  the  required  tasks  for  each  researcher  role. 

Training  of  new  personnel  in  experimenter  roles 

Additional  student  personnel  were  hired  to  supplement  experimenter  roles  during  participant  data 
collections.  Three  students  were  trained  to  serve  as  primary  and/or  secondary  researchers  for  each  of  the 
clinical  scenario  stations.  Training  was  administered  via  the  structured  learning  activities  used  to  train  other 
personnel  in  Quarter  2. 
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Distribution  of  procedure  feedback  to  study  participants 

It  was  identified  that  participants  in  our  study  desired  feedback  about  the  quality  of  their  performances 
after  participating  in  our  data  collection  sessions.  Giving  immediate  performance  feedback  at  data 
collections  would  lead  to  subjective  bias  in  how  participants  perform  other  stations,  so  it  was  decided  to 
give  them  analyzed  data  at  a  later  date  than  the  tests  were  performed. 

For  each  of  the  simulated  clinical  procedures,  information  was  analyzed  and  compiled  for  the 
quantitative  motion  data,  as  well  as  common  events  and  errors  observed  for  each  participant  and  the  group 
cohort.  These  metrics  about  their  performance  were  presented  to  participants  in  a  Microsoft  Excel-based, 
interactive  spreadsheet  interface  that  allowed  them  to  view  metrics  for  each  clinical  station  performed,  and 
compare  performance  to  their  peer  cohort.  Along  with  these  metrics,  this  Performance  Review  Tool 
interface  presented  videos  of  their  performance,  plots  of  their  motion  data,  and  images  of  their  final  product 
from  their  clinical  scenario.  The  tool  also  gave  participants  prescriptive  feedback  on  accepted  methods  for 
avoiding  some  of  the  common  errors  they  or  their  peers  committed  during  our  simulations. 

After  these  Performance  Review  Tools  were  distributed  to  study  participants  through  a  secure  online 
database,  we  held  four  focus  group  interview  sessions  with  past  and  future  participants  to  obtain  feedback 
about  the  tool.  Lessons  learned  from  these  focus  groups  indicated  which  sections  of  the  tool  were  found 
most  useful  and  in  which  areas  participants  desired  to  see  different  forms  of  information.  These  focus 
groups  were  essential  in  our  understanding  of  how  to  proceed  with  the  second  generation  of  this  software 
tool  for  distributing  relevant  and  useful  feedback  to  our  participants.  This  effort  also  aids  significantly  in 
the  retention  of  existing  and  recruitment  of  new  participants  for  the  study. 


Figure  21:  Performance  Review  Tool 
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Additional  Staffing  (Eran  Gwillim,  Caitlin  Jenewein ,  Becky  Ray) 

Three  additional  staff  members  have  been  added  to  the  research  team  over  the  course  of  the  last  year.  Each 
of  these  members  bring  specific  backgrounds  and  expertise  to  the  team,  which  will  overall  assist  with 
addressing  the  grant’s  objectives.  A  brief  description  of  each  new  member  as  well  as  their  role  in  the  project 
is  provided  below. 

Eran  Gwillim,  MD: 

Eran  Gwillim  received  his  MD  from  the  University  of  Illinois.  He  is  working  on  simulation  model  and 
station  design,  as  well  as  creation  of  performance  assessment  tools  and  analysis  of  research  study  participant 
performance. 

Caitlin  Jenewein: 

Caitlin,  BS,  was  hired  as  a  Research  Specialist  after  graduating  from  the  University  of  Wisconsin,  Madison. 
Her  continuing  roles  in  the  project  include  assisting  in  data  collection  and  analysis. 

Becky  Ray,  PhD: 

Dr.  Ray  received  her  receive  her  PhD  in  Psychology  and  Neuroscience  from  Stanford  University  and  is 
currently  working  towards  her  MS  in  Biostatistics  at  Pennsylvania  State  University.  Her  specific  role  in  the 
project  involves  conducting  statistical  analyses  and  assisting  in  the  development  of  data  analysis  protocols. 
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Key  Research  Accomplishments 

Table  1  below  provides  a  bulleted  list  of  the  project  accomplishments  as  organized  by  the  quarter  they  were 
achieved.  Additional  information  such  as  research  citations  and  specific  achievement  dates  have  been 
provided  whenever  they  were  available. 


Table  1.  List  of  Pro  ject  Accomplishments  per  Quarter 


1# 

Task 

Quarter 

Conducted  an  expert  review  of  each  of  the  four  clinical  procedure  stations,  which  included  an 

1 

audio  recorded  interview  and  demonstration  of  each  station  with  an  accomplished  attending 
physician  or  late-year  resident. 

One 

2 

Conducted  a  third  pilot  data  collection  and  updated  clinical  procedure  stations  to  address 
shortcomings,  inefficiencies,  and/or  data  collection  changes. 

One 

Created  and  updated  dedicated  “Station  Manager  Binders”  for  each  of  the  clinical  stations, 

3 

which  contains  all  of  the  relevant  information  for  correctly  setting  up  and  managing  the  station, 
inventorying  its  supplies,  and  recording  any  unexpected  incidents  relevant  to  the  integrity  of  the 
study  (e.g.  simulator  malfunction  or  participant  compliance  issues). 

One 

Designed,  built,  and  piloted  dedicated  station  fixtures,  which  have  been  custom  created  to 

4 

simplify  and  consolidate  each  of  the  clinical  stations  as  well  as  add  to  the  overall  professional 
aesthetic  of  the  stations. 

One 

5 

Organized  the  lab  personnel  into  Clinical  Station  Teams  whose  responsibility  is  to  be  trained  as 
an  “expert”  in  managing  one  to  three  of  the  four  clinical  procedure  stations.  Each  station  team 
was  trained  on  how  to  conduct  an  effective,  consistent,  and  objective  data  collection  while 
managing  each  of  the  clinical  procedure  stations.  The  team  members  were  then  each  assessed  on 
their  ability  to  perform  all  relevant  tasks  and  responsibilities  using  a  standardized  rubric. 

One 

6 

Divided  station  data  collection  responsibilities  into  two  categories  to  maximize  the  collection 
effort.  The  responsibilities  are  divided  between  running  the  motion  monitoring  software  and 
assisting  the  participant  in  the  collection;  along  with  minor  logistical  steps  imperative  to  the 
success  of  the  collection  time  of  twenty  minutes. 

One 

7 

Created  a  dedicated  station  cart  for  each  of  the  six  simulation  stations  as  well  as  a  general 
supplies/administrative  cart  and  a  motion-capture  cart. 

One 

8 

Created  cognitive  scenarios  for  participants  to  review  and  discuss  aloud  post  data  collection  on 
the  Central  Line  and  the  Urinary  Catheterization  stations. 

One 

Created  a  Performance  Feedback  Form,  which  will  provide  participants  with  an  interactive, 

9 

personal  report  on  their  performance  during  any  number  of  their  data  collection  sessions  or 
stations. 

One 

10 

Successfully  recruited  17  participants  to  represent  our  first  study  cohort. 

One 

11 

Finalized  the  “Station  Manager  Binders”  to  reflect  best  practices  including  a  more  detailed 
inventory  and  organization  to  the  mobile  units. 

Two 

12 

Finalized  the  Participant  Survey  Workbook  by  adding  additional,  clarifying  demographic 
questions. 

Two 

13 

Conducted  data  collection  from  38  residents  spanning  6  different  institutions.  Based  on 
experience,  a  total  of  37  participants  completed  the  data  collection. 

Two 

14 

Organized  post  collection  data  with  a  Post  Participant  Checklist  to  guarantee  that  all  data  was 
stored  in  the  same  method  and  confidentiality  was  ensured. 

Two 

15 

Updated  the  design  of  the  participant  feedback/report  card  model  to  best  fit  participant 
outcomes. 

Two 

16 

Dr.  Pugh  participated  in  the  Medical  Practice  initiative  IRP  in  Orlando,  FL  and  gave  a  verbal 
update  and  presentation  on  the  state  of  the  project.  She  discussed  both  the  accomplishments  and 
the  setbacks  of  the  project  and  was  well  received  by  her  peers. 

Two 
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Table  1.  List  of  Pro  ject  Accomplishments  per  Quarter^ 


1# 

Task 

Quarter 

17 

Continued  training  and  cross  training  additional  personnel  into  Clinical  Station  Teams  as  the 
needs  for  scheduling  the  data  collections  changed  based  on  locations  and  dates. 

Three 

Conducted  data  collection  from  an  additional  9  residents  from  the  University  of  Chicago, 

18 

bringing  the  total  from  year  one  of  data  collection  to  47  participants  (46  of  which  completed  the 
collection)  spanning  7  institutions. 

Three 

Purchased  Audio/Video  transcription  software  Transana  from  Wisconsin  Center  for  Education 

19 

Research  to  replace  MVTA  coding  program.  Began  to  train  staff  using  transcription  software  for 
continual  coding  and  categorizing  quantitative  and  qualitative  features  from  verbal  and  video 
recorded  data. 

Three 

20 

Distributed  the  Performance  Review  Tool  to  38  participants  while  the  final  data  analysis  was 
being  conducted  for  the  remaining  9  participants. 

Three 

21 

Two  abstracts  submitted  and  accepted  as  presentations  to  10th  annual  Academic  Surgical 

Congress  (ASC);  (Rutherford,  et  al,  2014;  Ray,  et  al,  2014). 

Three 

22 

Continued  data  analysis  for  further  conference  abstract  and  manuscript  submissions. 

Three 

23 

Began  communication  with  William  Lorie,  an  expert  in  analysis  and  design  of  survey 
assessments,  for  possible  collaboration. 

Three 

24 

Distributed  the  Performance  Review  Tool  to  remaining  9  participants  from  the  University  of 
Chicago  and  collected  feedback  on  the  tool  in  the  form  of  an  online  survey. 

Four 

25 

Dr.  Carla  Pugh  presented  the  talk,  “Skills  of  Hands  and  Mind:  Research  into  Decay,  Recovery 
&  Mastery”  at  The  International  Meeting  for  Simulation  in  Healthcare  meeting 

Four 

26 

Subcontractors  Felix  Huang  and  Sandro  Mussaivaldi  spent  2  days  in  Madison  (January  15th  and 
16th)  working  on  data  analysis  and  collaboration  time  line.  Weekly  teleconference  meetings 
continue  to  occur  with  them  both. 

Four 

27 

The  UW  Department  of  Surgery  Research  Summit  occurred  January  21st,  2015.  The  lab 
presented  5  posters. 

Four 

A  focus  group  was  held  at  ASC  Meeting  in  Las  Vegas,  NV  to  look  at  recruitment  efforts  and  the 

28 

Performance  Review  Tool  that  participants  were  sent  prior  to  the  meeting.  Meeting  occurred 
February  2nd-  February  5th,  2015  and  a  total  of  8  past  participants  from  4  different  site  locations 
attended  and  provided  feedback. 

Four 

Also  at  the  ASC  Meeting  in  Las  Vegas,  NV,  the  researchers  held  a  booth  in  order  to  generate 

29 

some  interest  in  recruitment  for  the  study.  Three  new  institutions  have  expressed  interest  and  are 
currently  in  contact  with  lab  management  about  coming  aboard  to  participate  in  the  study. 

Four 

30 

Two  more  focus  groups  were  held  at  the  UW  Madison  campus  for  additional  participants  that 
were  unable  to  attend  the  primary  focus  group  at  the  ASC  Meeting.  These  focus  groups  had  the 
same  agenda  points  and  were  held  on  February  17th  (one  in  the  morning,  one  in  the  afternoon). 

Four 

31 

A  data  collection  pilot  is  currently  being  planned  for  April  30th,  2015  and  protocol  updates, 
survey  updates,  as  well  as  station  enhancements  are  currently  underway. 

Four 

Recruitment  efforts  began  for  year  two  of  data  collection.  Several  institutions  have  expressed 

32 

continued  interest  and  will  be  meeting  with  Dr.  Pugh  at  the  ASE  Meeting  to  develop  a  plan  for 
championing  the  study  as  well  as  recruitment  efforts. 

Four 
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Reportable  Outcomes 

Table  2  provides  a  bulleted  summary  of  the  reportable  outcomes  achieved  over  the  past  year.  These 
outcomes  include  specific  methodological  improvements,  new  prototypes  such  as  the...,  and  also  specific 
products  such  as  conference  presentations,  research  papers,  assessment  instruments,  and  research  protocols. 
Specific  citations  have  been  provided  when  appropriate. 


Table  2.  List  of  Reportable  Outcomes  per  Quarter^ 


1# 

Type 

Outcome 

Quarter 

1 

Product 

Fully  refined  virtual  reality  stations  based  on  pilot  feedback. 

One 

2 

Methodology 

Refined  procedural  protocols  for  all  stations.  Collection-specific  materials  and 
instructions  created  for  motion  monitoring.  Trained  and  tested  team  in  proper 
procedure. 

One 

3 

Prototypes 

Fabrication  of  simulated  model  fixtures  to  ensure  data  collection  integrity. 

One 

A 

Product 

Created  station-specific  carts  stocked  to  create  smooth  setup  and  breakdown  as 

One 

‘-r 

well  as  address  transportation  needs. 

5 

Product 

Participant  workbooks  finalized. 

One 

6 

Product 

Improvements  implemented  in  motion  monitoring  software  as  well  as  video 
data  collection  setup. 

Two 

7 

Methodology 

Standardized  methods  developed  for  equipment  storage  and  transportation. 

Two 

8 

Product 

Training  documents  and  protocol  developed  for  sensor  application. 

Two 

9 

Methodology 

Data  archive  method  using  UW-Madison  REDCap  secured  database  developed 
to  expedite  data  entry. 

Two 

10 

Methodology 

Developed  methodology  for  analyzing  participant  workbook  and  survey  data 
and  obtaining  time  stamps  of  video  data  for  segmenting  regions  of  interest  in 
motion  data. 

Two 

11 

Product 

Completed  development  on  the  initial  version  of  the  Microsoft  Excel-based 
Performance  Review  Tool  to  send  to  participants.  Tool  contains  information 
about  their  individual  performance,  errors  observed  and  accompanying  media  of 
highlighted  video  clips. 

Three 

12 

Product 

Developed  survey  for  participants  to  assess  the  Performance  Review  Tool  to 
facilitate  future  enhancements  to  both  the  tool  and  the  study. 

Three 

13 

Product 

Two  presentations  were  accepted  and  presented  at  the  Academic  Surgical 

College  (ASC)  (Ray  et  al.,  2015;  Rutherford  et  ah,  2015). 

Three 

14 

Product 

One  ASC  presentation  was  accepted  as  a  manuscript  and  is  under  final  review  at 
the  journal  (Ray  et  ah,  2015,  under  review).  The  other  manuscript  is  in 
preparation  (Rutherford  et  ah,  2015,  in  preparation). 

Four 

Final  Conclusions 

Year  two  of  the  project  has  included  a  number  of  significant  steps  towards  meeting  the  four  key  study 
objectives  outlined  by  our  original  SoW.  Specifically,  the  team  has  successfully  completed  the  first  year  of 
data  collection;  refined  the  simulation  stations  for  the  initial  and  future  data  collections;  analyzed 
performance  on  multiple  stations;  and  disseminated  our  preliminary  work  in  the  form  of  several  papers  and 
presentations. 
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References  for  Abstracts  Submitted  as  Part  of  this  Project 
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